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Executive Summary 

The NASA Applied Sciences Directorate, part of the Earth-Sun System Division of NASA’s Science 
Mission Directorate, has partnered with the U.S. Department of Agriculture (USDA) to enhance decision 
support in the area of agricultural efficiency — an application of national importance. Key components of 
national agricultural decision making are to understand the global context of crop conditions, to 
understand how those conditions affect food security, and to predict global crop production of key 
commodities. The Production Estimates and Crop Assessment Division (PECAD) of the USDA Foreign 
Agricultural Service (FAS) accepted the mandate to support national decision making with timely global 
crop intelligence. PECAD accomplishes its mission with a proven set of decision support tools that 
facilitates both its own internal analyses and its dissemination of relevant information to policy makers 
and other stakeholders. 

This Verification and Validation (V&V) report is part of series that tracks the NASA/PECAD partnership 
through the systems engineering process. An Evaluation Report for PECAD was published in 2004 to 
document the decision support environment and to identify relevant NASA science assets that might 
improve the PECAD decision system. A Benchmarking Report was produced concurrent with this V&V 
report to measure the value of the initial NASA products that have been integrated into the PECAD 
system. This V&V report summarizes the Verification and Validation of the same enhancements covered 
in the Benchmarking Report: 

• Near-real-time Moderate Resolution Imaging Spectroradiometer (MODIS) products through 
PECAD’ s MODIS Image Gallery 

• MODIS Normalized Difference Vegetation Index (NDVI) time series data through the USDA- 
FAS MODIS NDVI Database 

• Jason- 1 and TOPEX/Poseidon lake level estimates through PECAD’s Global Reservoir and Lake 
Monitor (GRLM) 

Where possible, each enhanced product was characterized in terms of accuracy, timeliness, and coverage, 
and the characterized performance was compared to PECAD operational requirements ( the operational 
requirements do not necessarily match performance requirements for projects as funded). 

The accuracy of the MODIS Image Gallery NDVI product is being studied, but results are not yet 
available. The timeliness of the MODIS Image Gallery as measured by product latency was estimated to 
be 4.4 hours. Availability of MODIS Image Gallery products within the PECAD operational requirement 
of 6 hours was measured at 87 percent. In terms of coverage, the MODIS Image Gallery includes the 
great majority of crop production areas of economic importance but still lacks coverage in many areas 
where food security is a concern. 

PECAD’s requires that MODIS NDVI time series be consistent with the legacy of the Advanced Very 
High Resolution Radiometer and SPOT VEGETATION measurements. Studies indicate that such 
consistency has been achieved, and ongoing work shows good agreement among products as well. The 
latency of the USDA-FAS MODIS NDVI Database appears to be several days longer than the 72-hour 
PECAD requirement. USDA-FAS MODIS NDVI Database coverage of crop production areas of 
economic importance was all but exhaustive. 

The surface water level relative accuracy for the GRLM was estimated to be 5-7 cm over the Great 
Lakes, exceeding the 10-cm requirement established by PECAD. Some more challenging lakes (smaller 


v 


RELEASED - Printed documents may be obsolete; validate prior to use. 



Verification and Validation of NASA-Supported Enhancements to PECAD’s Decision Support Tools 


lakes, reservoir in rugged terrain) did not meet the 1 0-cm requirement but still arguably provided valuable 
information. In 2005, GRLM latency was estimated to be 6 days, which was a 10-day improvement over 
2004 performance. GRLM coverage was expanded, but because of challenges with current standard 
Jason- 1 data processing, most of the new water bodies in the GRLM have few, if any, valid height 
estimates in the Jason- 1 era. This leaves South America, south and east Asia, and Australia with very few 
near-real-time data points. 

As of su mm er 2005, NASA-supported enhancements to the PECAD decision support tools have had 
varying degrees of success in meeting PECAD’s requirements. The MODIS Image Gallery and the 
GRLM are more mature and have achieved a semi-operational status, whereas the USDA-FAS MODIS 
NDVI Database is still evolving and should presently be considered developmental. 

One overall recommendation resulting from the V&V process is that as NASA-supported enhancements 
become operational, regularly generated metrics should be introduced to track certain types of 
performance, such as delivery and coverage. While certain facets of decision support will require more 
careful analysis, the more routine aspects that relate to satisfaction of customer requirements should be 
amenable to simple self-monitoring that could be publicly available as product metadata. 
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1.0 Introduction 

NASA partners with the U.S. Department of Agriculture (USDA) to bring about innovative uses of 
NASA Earth-Sun system science to inform and improve decision making for the sake of U.S. agricultural 
efficiency. In this partnership, NASA and the USDA seek to identify assessments, tools, and systems that 
impact decision making at a national scale and that are amenable to enhancement with NASA 
observations or predictions. One decision support system (DSS) that has been identified belongs to the 
Production Estimates and Crop Assessment Division (PECAD). 

PECAD, a component of the USDA Foreign Agricultural Service (USDA-FAS), monitors global crop 
conditions, assesses how those conditions affect food security, and predicts crop yields for grains, 
oilseeds, and cotton. PECAD essentially performs an intelligence mission that affects U.S. government 
actions and global commodity markets. PECAD has several analysts who make crop assessments for 
certain regions of the globe. These analysts must make informed decisions about crop area, yield, and 
production. These decisions are supported by a framework of tools that bring together multiple, 
independent information sources in a spatial and temporal context. These tools include the Crop 
Assessment Data Retrieval and Evaluation (CADRE) geospatial data management system, which 
provides internal access to a wealth of remote sensing and weather information and to the Crop Explorer 
Web interface (USDA, 2005) which gives both internal and external users a portal to view remote sensing 
observations and model outputs for fixed geospatial subsets. Many of the Crop Explorer data streams are 
provided with near-real-time access. 

More detailed information about PECAD’s DSS can be found in Decision Support Tools Evaluation 
Report for FAS/PEC AD, Version 2.0 ( NASA, 2004a) . This report was part of a systems engineering 
approach to the USDA/NASA partnership supporting the enhancement of PECAD’s DSS. Three points of 
documentation are planned in this process: (1) DSS evaluation, (2) verification and validation of specific 
enhancement performance, and (3) benchmarking to show net effects of enhancements on the DSS 
performance as a whole. The systems engineering approach in its entirety is represented in Figure 1. This 
report documents the verification and validation (V&V) of implemented enhancements to PECAD’s DSS 
that are based on NASA Earth-Sun system science. 

The implemented enhancements include application-tailored Moderate Resolution Imaging 
Spectroradiometer (MODIS) products and Global Reservoir and Lake Monitor (GRLM) products. The 
MODIS products are currently made available through two separate decision support tools: the MODIS 
Image Gallery and the USDA-FAS MODIS Normalized Difference Vegetation Index (NDVI) Database. 
Both the GRLM and the MODIS Image Gallery provide near-real-time products through PECAD’s 
Crop Explorer. This discussion will address two areas: 

• Assessments of existing or previous NASA products on which these enhancements are based, and 

• Characterizations of the performance of the newly implemented products. 

1.1 Systems Engineering Considerations 

While there is an overall thematic direction to the systems engineering process as defined in Figure 1, the 
process is not a simple sequence. For example, V&V and benchmarking for PECAD’s DSS have 
proceeded concurrently with the V&V effort coordinated by NASA Stennis Space Center’s Applied 
Sciences Directorate and benchmarking carried out by a joint team of scientists from the University of 
Arizona and the University of Missouri (Haithcoat et ah, 2005: van Leeuwen et ah, 2005) . 
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DSS 

Selection 


Evaluation V&V Benchmarking 

Systems Engineering Approach 

A systems engineering approach leads to scalable, systemic, and sustainable 
solutions and processes that contribute to the success of the mission, goals, 
and objectives of each National Application. 


Source: Adapted from Bahill and Gissing 1998 

Figure 1 . Systems engineering approach. 

V&V and benchmarking milestones’ being reached does not imply that the systems engineering process is 
complete. This certainly applies to the design and implementation stage. The projects currently delivering 
both MODIS and GRLM products are at various levels of maturity and are only meant to attain semi- 
operational or perhaps even demonstration status. Since the PECAD DSS certainly requires fully 
operational solutions, a transition to operational status is inferred. The manner and schedule of the 
operational transition is a matter of ongoing dialogue among DSS stakeholders. 

The distinction between pre-operational and operational status has important consequences in the 
understanding of DSS requirements. This V&V report and its predecessor evaluation report (NASA, 
2004a) consider the operational requirements of PECAD’s DSS. The projects implementing enhanced 
products are properly focused on their project performance requirements as defined in agreements with 
NASA and the USDA. Where a clear separation between operational and project performance 
requirements is known to exist, this separation is noted in the report. 

Simply defining PECAD’s operational requirements has been a difficult process, and the process is 
ongoing. The challenges faced by PECAD’s analysts are diverse and resistant to simplification. Spatial 
scale ranges from small subsistence plots to the entire crop-producing area of the globe. Temporally, 
analysts must track multiyear means and must also be able to predict agronomic damage only hours after 
an event occurs. As a result, some requirements are a matter of continuing discussion between PECAD 
and NASA. Specifications for the spatial characteristics of Earth science inputs fall into this category. 
PECAD uses remote sensing data with ground sample distances (GSDs) ranging from less than a meter 
up to multiple kilometers. Teasing out spatial requirements for PECAD’s DSS would necessitate a careful 
breakdown of analysts’ typical tasks followed by a detailed sensitivity analysis for each task. Even then, 
trade studies would be required to balance spatial considerations with information technology and 
resource constraints and with temporal drivers. 
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The operational requirements suitable for attention in this report fall into three areas: 

• Accuracy: measured uncertainty of a particular product’s parameter of interest 

• Delivery: 

- Data time step, or frequency of product delivery 

- Latency, or mean lag time between original data acquisition and final product delivery 

• Coverage: geographical area covered or spanned by the product 

The requirements as specified for each product are taken from the earlier evaluation report (NASA, 
2004a) unless otherwise noted. 

2.0 V&V of Application-Tailored MODIS Products 

As mentioned above, PECAD’s overall set of decision support tools are enhanced by the routine 
provision of two application-tailored MODIS products: the Crop Explorer MODIS Image Gallery and the 
USDA-FAS MODIS NDVI Database. Both products have been developed and implemented by the 
Global Agricultural Monitoring (GLAM) project, whose collaborators include NASA’s Goddard Space 
Flight Center (GSFC); USDA-FAS; Science Systems and Applications, Inc.; and the University of 
Maryland Department of Geography. 

MODIS Image Gallery: The Crop Explorer MODIS Image Gallery is accessed through the Crop 
Explorer interface at the following USDA-FAS Web site: 
http://www.pecad.fas.usda.gov/cropexplorer/modis summary/ 

The Crop Explorer MODIS Image Gallery provides near-real-time access to processed MODIS image 
subsets. This interface is actually a thin client linked to a selected set of the imagery available through 
GSFC’s MODIS Rapid Response System. For each image subset, the following images are provided in 
JPEG format: a natural color combination (Red, Green, Blue (RGB): MODIS bands 1, 4, and 3); a 
shortwave infrared/near infrared/red combination (RGB: MODIS bands 7, 2, and 1); and an NDVI. 
GeoTIFF format is provided at many locations and is planned for all image subsets as hardware 
improvements permit (Descloitres, personal communication, 2005). Each image type may be downloaded 
in 250 m, 500 m, and 1 km ground sample distances. Accuracy, delivery, and coverage requirements 
related to near-real- time MODIS images are detailed in Table 1. 


Table 1 . PECAD accuracy, delivery, and coverage requirements for near-real- time imaging. 


Category 

Requirement 

Product accuracy 

Undetermined 

Data time step 

1 day 

Latency 

6 hours or less 

Coverage 

All land regions important for crop production and food security 


Source: NASA, 2004a 


Further comment is called for in the case of product accuracy and coverage requirements. For product 
accuracy, a need for consistency with multiday composite products (such as the USDA-FAS MODIS 
NDVI Database) is implied, but no requirements or specifications have been established in discussions 
with PECAD. The MODIS Rapid Response team is studying the accuracy of the Rapid Response NDVI 
relative to other estimates, but results are not yet available (Descloitres, personal communication, 2005). 
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The coverage requirement is a case where operational and project performance requirements diverge. 

That PECAD needs imaging over “all land regions important for crop production and food security” 
(NASA, 2004a) is self-evident, but the MODIS Rapid Response contribution as originally proposed did 
not specify coverage. The project focus was on generating prototyping products (Justice et al., 2002) . In 
early 2005, PECAD and the MODIS Rapid Response team agreed to bring the project performance 
requirement into alignment with PECAD’s quasi-global operational requirement. Contingent on the 
procurement and integration of new hardware to support this effort, the MODIS Rapid Response team 
expects to complete the increase in coverage by the end of the 2005 calendar year (Kendall and Schmalz, 
personal communication, 2005). 

USDA-FAS MODIS NDVI Database: While the MODIS Image Gallery meets near-real-time needs, 
PECAD also requires a quasi-global NDVI image and time series database to observe crop trends. 
Multiday compositing is necessary for this product to reduce the affect of clouds, bidirectional reflectance 
distribution function effects, and other confounding factors. Through the USDA-FAS MODIS NDVI 
Database, users may display or download the archived NDVI data as single-date images or as chart time 
series of NDVI variation for selected points or areas. This database is currently maintained for PECAD at 
the following GSFC Web site: http://pekko.geog.umd.edu/usda/ndvi/ 

This product is still a beta version. The NDVI observations in this product are not produced by the 
GLAM project; they are NDVI estimates computed from standard Level 3, 16-day surface reflectance 
composites (MODIS MOD09 product) generated by the MODIS Data Production System (MODAPS). 
The USDA-FAS MODIS NDVI Database is developing tools for MODIS NDVI products that should 
play a similar role to Advanced Very High Resolution Radiometer (AVHRR) vegetation index 
information currently available to PECAD analysts through CADRE. The PECAD accuracy, delivery, 
and cover requirements for an NDVI database are summarized in Table 2. 


Table 2. PECAD accuracy, delivery, and coverage requirements for an NDVI database. 


Category 

Requirement 

Product accuracy 

Must be consistent with existing NDVI database going back to 1981 

Data time step 

Maximum of 10 days (Doom. 2004) 

Latency 

24 hours or less measured from the end of the compositing period (Doom. 2004) 

Coverage 

All land regions important for crop production and food security 


Source: NASA, 2004a 


The USDA-FAS MODIS NDVI Database contains points of difference between operational and project 
performance requirements. For the data time step, the product as proposed is limited to 16-day 
compositing periods, which are the minimum available through MODAPS. Similarly, the product is also 
limited in latency by the performance of MODAPS. The project’s original proposal suggested a nominal 
latency of 7 days. 

2.1 Calibration and Validation of MODIS Sensor and Standard Products 

The on-orbit calibration and validation of the MODIS sensor and standard MODIS products was dealt 
with comprehensively (for Terra MODIS) in a special issue of Remote Sensing of Environment in 2002, 
which characterized several intrinsic properties of interest to operational users. Wolfe et al. (2002) 
reported a geolocational accuracy of 50 m (1 a) at nadir. For MODIS Bands 1 and 2 (the red and near- 
infrared bands used for PECAD tailored products), Rojas et al. (2002) found that the spatial response was 
consistent with pre-launch performance in the cross-track direction but was a bit degraded in the along- 
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track direction. Huete et al. (2002) demonstrated that MODIS NDVI performed at higher fidelity than 
AVHRR NDVI. In a discussion of atmospheric correction for standard MODIS products, Vermote et al. 
(2002) showed a clear example of how not correcting for aerosol can dramatically affect the distribution 
of NDVI values throughout a scene. 

Further work has been done regarding the cross validation of MODIS NDVI with other sensors. Gallo et 
al. (2004) found a linear relationship between AVHRR and MODIS 16-day composite time series with 
overall coefficient of variation (r 2 ) above 0.9 (considering nine diverse land cover types). Working 
directly in support of the NASA/FAS partnership, the Global Inventory Monitoring and Modeling Studies 
(GIMMS) group at NASA GSFC has found similar results studying AVHRR, MODIS (MOD 13 standard 
products), and SPOT VEGETATION (Justice, 2005) . In Figure 2, NDVI time series are displayed from 
GIMMS estimates for one crop production region in South Africa and for another region in eastern 
Australia (the green line in each plot is MODIS Enhanced Vegetation Index, which is not used in current 
MODIS products tailored for PECAD). All of the NDVI time series have correlation coefficients above 
0.9 relative to each other. 




Figure 2. Vegetation index time series for the Maize Triangle region of South Africa and for the Victoria wheat- 
growing region of eastern Australia. 


2.2 Characterization of the MODIS Application-Tailored Products 

The product characterizations in this report summarize quantifiable product performance in the areas of 
accuracy, delivery, and coverage as outlined in the systems engineering discussion (Section 1.1). 

2.2.1 MODIS Product Accuracy 

The only parameter of interest that has been clearly established for MODIS application-tailored products 
is NDVI. Given that PECAD’s accuracy requirement has been stated only in qualitative terms 
(observations should be “consistent” with the multiyear database), only a cross-comparison of the 
application-tailored products to standard products is needed. 
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A cross-comparison for the MODIS Rapid Response NDVI is not yet complete. The MODIS Rapid 
Response System team is currently undertaking a comparison with standard MODIS products 
(Descloitres, personal correspondence, 2005). 

Because NDVI in the USDA-FAS MODIS NDVI Database is actually computed by MODAPS, it is 
assumed to be validated for the purposes of this report. It should be noted, however, that the USDA-FAS 
MODIS NDVI Database product is not the standard MOD 13 NDVI; this NDVI product is computed in 
the MOD44C vegetation intermediate composite product. Although MOD44C is primarily an 
intermediate step in production of the MOD44A vegetation cover conversion product, its origins at the 
University of Maryland give the GLAM team greater creative control in support of PEC AD. Both 
MOD 13 and MOD44C take daily MOD09 surface reflectance as input and both use the standard equation 
( (Pvffl _ P,ed )/ {Pnir + P red ) ) t0 compute NDVI, but the two slightly differ in their compositing 
schemes. MOD44C seeks the most cloud free, highest resolution data over the compositing period 
(USGS, 2005) , and MOD 13 has a similar approach called “constraint view angle maximum value 
compositing” which diverges in some details (Gao et al., 2003) . Given the product similarities, previous 
studies regarding consistency of MODIS with legacy systems should be considered extensible to USDA- 
FAS MODIS NDVI Database estimates. 

2.2.2 MODIS Product Delivery 

The MODIS Rapid Response System product delivery was observed for two 15-day periods. The first 
period ran from June 23, 2005, (Julian day 173) through July 6, 2005 (Julian day 187). However, on July 
2, 2005, an interruption of data lasted for more than 2 days. This problem affected all the data that was 
passed through the MODIS Rapid Response System to the Crop Explorer MODIS Image Gallery and was 
reported on the MODIS Rapid Response System Status page as shown in Figure 3. 


Date 

Description 

2 July 2005 
2005/183 

Some current data is not available due to hardware issues. 


Source: http://rapidfire.sci.asfc.nasa.aov/status/ 

Figure 3. July 2005 data interruption as reported on the MODIS Rapid Response System Status page. 


Even 5 days after the July 2, 2005, interruption, the MODIS Rapid Response Delivery System delivery 
times were notably out of step with pre-interruption system performance. As a result, only the first 10 
days of observation were used to estimate system performance. Based on the June- July observations, the 
estimated system latency for 59 tiles with an Aqua and a Terra delivery opportunity for each of 10 days 
(1180 distinct delivery opportunities) was 4.7 hours. The latency was computed as the mean lag time. All 
lag times greater than 24 hours were excluded on the assumption that these times represented anomalous 
problems that were not typical of system throughput. Just over 65 percent of the MODIS Rapid Response 
System information was available within the 6-hour requirement stated by PECAD and 80 percent of the 
information was available within 24 hours. 

The second observation period ran from October 1, 2005, (Julian day 274) through October 15, 2005 
(Julian day 288). Although 12 tiles were added to cover North Africa before the October observation 
period, these tiles were left out of this analysis to allow valid comparisons between the two periods. 
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Based on the October observations, the estimated system latency for 59 tiles with an Aqua and a Terra 
delivery opportunity for each of 15 days (1770 distinct delivery opportunities) was 4.4 hours. About 
87 percent of the MODIS Rapid Response System information was available within the 6-hour 
requirement stated by PECAD and 1 00 percent of the information was available within 24 hours. 

Given the July data interruption, the October period should be considered more representative of expected 
operational system performance. Several distinct trends can be discerned in the October period. First, it 
should be noted that the lag times are heavy-tailed on the long side (Figure 4). The observed lag time data 
bears some resemblance to heavy-tailed distributions such as the inverse Gaussian and lognormal 
distributions. The implication is that for mature requirements it will be important to develop thresholds 
for both mean delivery and for some higher threshold with 90 percent delivery. 


A systematic relationship existed 
between lag time and the time of 
day in which a particular subset- 
tile was acquired and processed. 
Figure 5 shows that the lag time 
trend reaches a minimum of about 

3 hours near 6:00 AM - 12:00 
AM (UTC) and then rises slowly 
through the course of a day, 
reaching a maximum of more than 

4 hours between 6:00 PM and 
midnight (UTC). The geographic 
distributions of delivery and lag 
times for Terra and Aqua are 
shown in Figure 6. 



Figure 4, Distribution of Crop Explorer MODIS Image Gallery 
delivery time lag from October 1, 2005, through October 15, 2005. 
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Figure 5. Variation of Crop Explorer MODIS Image Gallery delivery time lag by time of day 
(October 2005 observation period). 
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Taken as a whole, the Crop Explorer MODIS Image Gallery latency is meeting the stated PECAD 
requirement of 6 hours. Nominally, the data time step is being met, but lapses in availability (both 
identified and unidentified in the MODIS Rapid Response System Status page) are a cause for concern. 

The USDA-FAS MODIS NDVI Database was not monitored in as careful a manner because it is more 
developmental in nature. However, it was observed that through June and July 2005, the shortest time 
lapse between end of compositing window and USDA-FAS MODIS NDVI Database timestamp was 
about 9.5 days. This latency has not yet met the project goal of 7 days nor the PECAD goal of 72 hours as 
stated in the DSS evaluation (NASA, 2004a) . Given the 16-day time step and a latency of about 10 days 
(or more), the chance of PECAD analysts having more than one NDVI Database reading for the month 
leading to lock-up is small. 



Figure 6. Geographic distribution of Crop Explorer MODIS Image Gallery delivery and lag times 
for Terra and Aqua (October 2005 observation period). 


2.2.3 MODIS Product Coverage 

The global footprints of the Crop Explorer MODIS Image Gallery and the USDA-FAS MODIS NDVI 
Database are shown in Figure 7. Curiously, the Database omits Madagascar, even though Madagascar is 
available through the Crop Explorer MODIS Image Gallery. However, the USDA-FAS MODIS NDVI 
Database coverage limitations are strictly a result of the current system configuration since the MO DAPS 
source data is essentially global in coverage. 

The Crop Explorer MODIS Image Gallery omits a few crop production areas of some economic 
importance including Central America, parts of South America, the northernmost productive region of the 
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Great Plains of North America, and parts of central Asia. Note the presence of the 12 northern Africa 
subset-tiles for the Crop Explorer MODIS Image gallery that were left out of the delivery time analysis. 
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Figure 7. Comparison of Crop Explorer MODIS Image Gallery and MODIS NDVI Database 
geographical footprints as of October 2005. The footprints are overlaid on a global map of 
croplands as estimated in the 2001 MODIS Land Cover product (MOD12). 


Overall, the MODIS application-tailored products have nearly met the PECAD coverage requirements for 
economically important crop production, but important coverage gaps exist in the Crop Explorer MODIS 
Image Gallery in terms of PECAD’s food security mandate: notably Central America, northern South 
America, Indonesia, and the Philippines. 

3.0 V&V of Global Reservoir and Lake Monitor Products 

PECAD’s decision support requirements related to surface water level variation are detailed in Table 3. 


Table 3. PECAD accuracy, delivery, and coverage requirements for surface water 
level variation. 


Category 

Requirement 

Surface water level relative accuracy 

10 cm 

Data time step 

1 0 days 

Latency 

7-14 days 

Coverage 

Surface waters in all land regions important 
for crop production and food security 


Source: NASA, 2004a 


3.1 Calibration and Validation of TOPEX/Poseidon and Jason-1 Radar Altimeters 

Global Reservoir and Lake Monitor products are based on measurements from radar altimeters onboard 
the Jason- 1 and TOPEX/Poseidon satellites (USDA, 2004) , which are joint missions between NASA and 
the French space agency, the Centre National d’Etudes Spatials (CNES). The primary purpose of these 
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altimeters is to increase understanding of ocean surface topography. Their principal measurement is sea 
surface height (Menard et al., 2003) . The ranging approach for determining sea surface height differs in 
some particulars from the approach for determining surface water levels, but the two measurements are 
based on many common principles. Given that Jason- 1 and TOPEX/Poseidon were designed for open sea 
measurements, the accuracy achieved by these instruments in measuring sea surface height sets the 
optimistic bound on accuracy that can be expected in measuring surface water levels. 

After the launch of Jason- 1 on December 7, 2001, about 10 months were dedicated to calibration of the 
new instrument and assessment of its performance. For much of that period, the Jason- 1 and 
TOPEX/Poseidon satellites were operated in a common orbit with about one minute of separation 
between the two platforms, allowing an updated assessment of the TOPEX/Poseidon performance as well 
as a cross-validation between the two systems. 

Three primary levels of distributed products were evaluated: 

1. Geophysical Data Record (GDR) - final product based on ground processing, available in 30 days; 

2. Interim Geophysical Data Record (IGDR) - intermediate products based on a preliminary orbit 
determination, available in three days; and 

3. Operational Sensor Data Record - near-real-time products based on onboard processing, available in 
3-5 hours. 

Of these products, the GDRs and IGDRs are relevant to the Global Reservoir and Lake Monitor. The 
GRLM used GDRs for TOPEX/Poseidon data to establish historical data going back to 1992 and used 
IGDRs for Jason- 1 data to provide a near-real-time product with a latency period on the order of 10 days. 

Menard et al. (2003) summarized the Root-Sum-of-Squares estimates of Jason- 1 sea surface height errors 
as 3.3 cm for the GDR product and as 3.9 cm for the IGDR products. Working in diverse locations, 
including the Harvest platform near California (Haines et al., 20031 , Corsica (Bonnefond et al., 20031 , and 
the Bass Strait on the north side of Tasmania (Watson et al., 20031 , the individual calibration/validation 
teams commissioned by CNES and NASA all achieved sea-based results consistent with the overall 
estimates. Each of these studies also measured TOPEX/Poseidon errors and found them to be in the same 
3-4 cm range. Chambers et al. (2003) cross-validated Jason- 1 against TOPEX/Poseidon. The estimates of 
individual sensor error were similar among the analyses cited. The magnitude of the global bias between 
the sensors was found to be 14-15 cm. Residuals showed dependency on significant wave height, with 
local biases near coastlines. More placid waters had somewhat lower biases, and local biases in rougher 
waters were somewhat higher. This dependency on significant wave height accounts for some differences 
in bias estimates at the fixed sites, and it has implications for surface water level estimates since surface 
waters frequently have smaller wave heights than the open ocean. 

Further highlighting the challenge of altimetry over surface waters, one Jason- 1 calibration/validation 
effort studied sea surface height as measured over the Great Lakes with a special focus on a specific 
calibration site on the Ohio shore of Lake Erie (Shum et al, 20031 . After accounting for instrument bias, 
the Jason- 1 residuals with Great Lake tidal gauges ran between 3 and 5 cm, which is consistent with other 
results. However, the relative bias between Jason- 1 and TOPEX/Poseidon was estimated to be about 
8 cm, which is 6 cm less than the global bias. Only a small portion of the difference could be attributed to 
wave height effects. Ultimately, the study could not explain the difference. 
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An extensive evaluation of the capability of 
TOPEX/Poseidon to monitor lake/reservoir 
water level variation was carried out by 
Birkett (1995) . Birkett’s study included an 
analysis of 68 lakes and developed an error 
budget for surface water level variation as 
detailed in Table 4. In this budget, two factors 
stood out as creating differences between large 
and small lakes: range precision and wet 
tropospheric correction. 

Range precision was significantly degraded by 
freezing of the lake surface. Also, range 
precision varied according to the number of 
data points collected across a lake. As the 
number of data points per pass (N) over a 
lake was increased, the impact of range 
precision error was reduced by a factor 
of 1 / y[N . Birkett noted that the range 
precision error contribution was in the sub- 
centimeter range for large water bodies. 


Table 4. Total error budget for lake height. 


Contribution 

Root Mean Square 
(cm) 

NASA orbit 

3.5* 

Range precision 

(variable) 

Electromagnetic bias plus 
skewness 

2 + 1.2 

Earth tide 

1 

Dry troposphere 

0.7 

Wet troposphere (TMR/ECMWF 1 ) 

3/1.2 

Ionosphere (DORISVCNES) 

1.7 

Minimum total (TMR/ECMWF) 

4.85/5.57 


Source: adapted from Birkett. 1995 
* The removal of geographically correlated orbit error components via 
the use of repeat track techniques may reduce this value. 
f TOPEX Microwave Radiometer/European Centre for Medium-Range 
Weather Forecasts 

: Doppler Orbitography and Radiopositioning Integrated by Satellite 


The wet tropospheric correction was based on the onboard microwave radiometer for open ocean 
readings. However, because of the large footprint of the radiometer (20-40 km), the corrections based on 
microwave radiometer readings near shore could not be used. Thus, larger lakes benefited from a better 
correction than the smaller lakes. 


Aside from considerations related to the generalized error budget, Birkett also drew attention to several 
other conditions that could lead to error in estimation of surface water level variation. These conditions 
included unusually stormy or calm lake surfaces, unidirectional wind setup, and geoid surface slope error. 

Birkett’s study performed a validation versus gauge data for Lake Superior, Lake Michigan, and Lake 
Ontario and, consistent with Shum et al. (2003), found accuracy in the 3-5 cm range. 

3.2 Characterization of the Global Reservoir and Lake Monitor 


An initial verification and validation effort for the GRLM decision support tool was carried out in 2004. 
Though somewhat early in the implementation phase for the tool, this effort touched on all the 
requirements mentioned in Table 3. The tool’s delivery and coverage have matured since this initial V&V 
effort ended, so requirements related to these properties were re-evaluated through the month of May 
2005. As for the product accuracy requirement, the fundamental algorithm is essentially unchanged, so 
the results from the 2004 report are summarized below. Lor further detail, refer to the online version of 
the previous report, PECAD ’s Global Reservoir and Lake Monitor: A Systems Engineering Report 
(NASA 2004b) . 

3.2.1 GRLM Accuracy 

The GRLM root mean square accuracy estimates over several North American lakes are summarized in 
Table 5. These estimates are based on comparisons with water level gauges. The Great Lakes gauges were 
chosen from U.S. National Oceanic and Atmospheric Administration and Canadian Department of 
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Fisheries and Oceans gauges to frame the Jason-1 pass used for GRLM estimates (Figure 8). Lake 
Winnebago and Lake of the Woods were each referenced to a single U.S. Geological Survey gauge. Lake 
Powell was monitored by the U.S. Bureau of Reclamation using a well-gauge built directly into the face 
of Glen Canyon Dam. Essentially, the Great Lakes can be considered representative of the larger water 
bodies in the GRLM. Great Lakes errors ran between 5 and 7 cm. Lake Winnebago and Lake of the 
Woods may be somewhat representative of smaller water bodies. Each estimate represents only a handful 
of potential readings per pass. Lake Winnebago error was calculated to be 27 cm and Lake of the Woods 
error was computed to be 26 cm. Lake Powell represents the most challenging type of lake or reservoir. 
The lake has very few potential readings per pass, and it is surrounded by mountainous terrain. Lake 
Powell error over the period assessed was about 140 cm. 

Note the Great Lakes errors computed for the GRLM are somewhat larger than those reported by Shum et 
al. (2003) . This discrepancy may be explained by the earlier study’s focus on the verification of sensor 
performance and by corrections for the sea surface height estimates being carefully tailored for the Great 
Lakes. The 2004 GRLM verification effort made no special corrections to the data because the focus of 
that study was the GRLM operational product accuracy. 


Table 5. Verified GRLM performance over select North American surface water bodies.* 


Water Body 

Surface 

Area* 

(km 2 ) 

CT 

(outliers 

removed) 

(cm) 

Number 

Passes 

Outlying 

Passes 

Removed 

Number 

Gauges 

Mean Pass- 
to-Gauge 
Distance 
(km) 

Lake Superior 

82,100 

7.3 

81 

0 

3 

72 

Lake Michigan 

57,800 

5.3 

75 

0 

3 

51 

Lake Huron 

59,600 

6.2 

80 

2 

4 

55 

Lake Erie 

25,700 

6.5 

73 

1 

4 

29 

Lake Ontario 

18,960 

4.5 

79 

3 

3 

52 

Lake Winnebago 

557 

27.0 

24 

0 

1 

7 

Lake of the Woods 

1,900 

26.1 

42 

1 

1 

31 

Lake Powell 

500 

140.6 

244 

6 

1 

84 


* All lakes and reservoirs were characterized using Jason-1 IGDR-based estimates with the exception of Lake Powell. No estimates 
based on Jason-1 data existed for Lake Powell as of May 2004, so TOPEX/Poseidon GDR-based estimates were used. 
f Surface areas from www.worldlake.org . 


Note the Great Lakes errors computed for the GRLM are somewhat larger than those reported by Birkett 
(1995) or Shum et al. (2003) . Much of the increased Great Lakes error is likely due to ice contamination, 
storms, or wind setup. The 2004 GRLM validation effort attempted to screen data only by removing a 
small number of statistical outliers and, in the case of Lake Winnebago and Lake of the Woods, by 
excluding completely frozen conditions. The goal was to represent the unvarnished operational product 
accuracy. 

More problematic, perhaps, are the larger error figures for the three smaller water bodies: Lake 
Winnebago, Lake of the Woods, and Lake Powell. These error figures can be partially explained by less 
ideal reference data. As mentioned above, each of these three water bodies was referenced to only a single 
gauge. Lake Winnebago and Lake of the Woods estimates were based on fewer passes (24 and 42, 
respectively). The Lake Powell gauge would be expected to be markedly influenced by local draw-down 
associated with release events. Beyond reference issues, each of the three lakes had much less surface 
area than even the smallest of the Great Lakes. Consequently, the three lakes had fewer data points per 
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pass and reduced range precision relative to the Great Lakes. The three lakes suffered from poorer 
estimation of the wet tropospheric correction. The steep terrain surrounding Lake Powell made it difficult 
for the altimeter sensors to “lock” on the water surface as the satellite crosses from land to water. “Loss of 
lock” can cause a loss of some or all data points for surface water bodies. No valid data points have been 
returned for Lake Powell since the transition from TOPEX/Poseidon to Jason- 1. 


A Great Lakes Area Gauge Stations 



O Canada DFO Gauge SQticris 
Q US/NOAA Gauge Stations 
^meter Gx>und Track 


Recorded S*at Poirt 
• Recorded Bid Point 


Figure 8. Gauge stations in the general vicinity of the Great Lakes. 


While not all lakes appeared to meet the stated FAS accuracy requirement, perhaps the accuracy 
requirement should be revisited. For deep, narrow reservoirs, fairly large height uncertainty may be 
acceptable because the information regarding water volume is still readily discemable. For shallow, 
closed lakes (e.g., Fake Chad), the current standards are appropriate. Furthermore, FAS lake water height 
requirements should be developed in more detail to include a comprehensive listing of water bodies that 
FAS believes should be monitored. 

3.2.2 GRLM Delivery 

The latency of the GRFM products decreased from 10.3 days to 5.8 days between May 2004 and May 
2005. Fatency was defined as the mean lag time between data acquisition and GRFM product upload. The 
maximum lag time in May 2004 was 19.2 days, but in May 2005 the maximum lag time was reduced to 
10.4 days. This dramatic improvement in delivery time was accomplished by reducing the average time 
between GRFM updates from 16 days to 6 days. This improvement clearly meets the stated FAS 10-day 
latency requirement for surface water level variation. 
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Once every 9.9 days (the Jason-1 orbital period), the GRLM adds a new data record for each monitored 
body of water, so the FAS 10-day time step requirement is nominally met. Flowever, the Jason- 1 standard 
product algorithms yield no height information for some passes, perhaps because of issues like “loss of 
lock” or other conditions that are screened in the standard algorithms to improve sea surface height 
estimates. For many instances in both 2004 and 2005, months to years passed between data acquisitions 
in which water levels were actually estimated. These dated sites have archival value, but they no longer 
provide information about current regional water availability. A breakdown of the time to last estimated 
water level is shown in Figure 9. 



Figure 9. Histogram of time to last estimated water level checked at the 
end of May 2004 and again at the end of May 2005. 


From May 2004 to May 2005, the number of lakes and reservoirs monitored increased from 52 to 84 - 
more than 60 percent. An incremental increase occurred in the number of water bodies with near-real- 
time monitoring, but most of the increase consisted of lakes and reservoirs that have had little or no valid 
data since Jason- 1 became operational. The dearth of data for near-land sites results from the algorithms 
implemented for standard Jason- 1 data processing. These algorithms filter out the near-land data. This 
issue was noted in the 2004 report (NASA, 2004b) with the indication that it was being addressed, but, as 
of this writing, no substantial improvement in processing near-land data was noted. The geographic 
distribution of more current and more out-of-date sites is shown in Figure 10. 
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Figure 10. Geographic distribution of time to last estimated water level at the end of May 2005. 


3.2.3 GRLM Coverage 

In 2004, little coverage existed in South America, western North America, or eastern Asia. Australia had 
no monitored sites. As can be seen in Figure 10, through May 2005 sites have been added in all of these 
areas. Unfortunately, the overwhelming majority of the new sites are badly dated, so the near-real-time 
coverage has changed little. 

4.0 Conclusions and Recommendations 

As of summer 2005, NASA-supported enhancements to the decision support tools of PECAD have had 
varying degrees of success in meeting PECAD’s requirements. The MODIS Image Gallery and the 
Global Reservoir and Lake Monitor are more mature and have achieved a semi-operational status, 
whereas the USDA-FAS MODIS NDVI Database is still evolving and should presently be considered 
developmental. 

Considering each decision support enhancement individually, their performances can be summarized 
according to product accuracy, delivery, and coverage. The MODIS Image Gallery has met delivery 
requirements with a latency of less than 5 hours and a time step of 1 day. Down time is a concern. 
Coverage is nearing sufficiency for economically important crop production regions, but some significant 
gaps remain in Central America, in parts of South America, in the northernmost productive region of the 
Great Plains of North America, and in parts of central Asia. Additional large coverage gaps for PECAD’s 
food security mission exist, notably in the Congo basin, in Indonesia, and in the Philippines. 

Although the USD A FAS MODIS NDVI Database is less finished than the MODIS Image Gallery, the 
Database has already largely met its coverage requirement with fewer gaps than the MODIS Image 
Gallery. The principal exception is the Congo basin, where the Database does not seem to be meeting the 
72-hour latency requirement. Cursory observations indicated latency of 9 days or more. The Database is 
being delivered on a 16-day time step, which is at the upper end of the range currently considered 
acceptable by PECAD. PECAD’s final time step requirement is a matter of ongoing discussion. The 
consistency of standard MODIS NDVI products with the AVHRR NDVI products has been established. 
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The Global Reservoir and Lake Monitor exceeded PECAD delivery requirement with a latency of less 
than 6 days and a time step of 10 days. Validation with water level gauges showed that the 10 cm PECAD 
surface water level relative accuracy requirement was being met in the Great Lakes. Results were not as 
good over smaller water bodies, which can be partially explained by less ideal reference data. While not 
all lakes appeared to meet the stated FAS accuracy requirement, perhaps the accuracy requirement should 
be revisited given that even “noisier” data often seemed to satisfy underlying information needs. 

The amount of data that is not being processed by the standard Jason- 1 algorithms was established as a 
matter of concern in 2004 and continues to be a problem. Coverage improved from 2004 to 2005, so that 
lakes and reservoirs being monitored are approaching quasi-global status, but the overwhelming majority 
of the added lakes have had no valid heights estimated because of the problems with the standard Jason- 1 
algorithms. Data dropouts should be reduced. At present, the source of science data for the GRLM is the 
“Archivage, Validation et Interpretation des donnees des Satellites Oceanographiques” (AVISO) data 
center for CNES. If AVISO is unable to improve the data stream, perhaps a separate channel for 
providing appropriate sea surface height data should be engaged at the Jet Propulsion Laboratory’s 
Physical Oceanography Distributed Active Archive Center. If no action is forthcoming, then the need 
should be elevated to higher levels within NASA. 

One overall recommendation resulting from the verification and validation process is that as NASA- 
supported enhancements become operational, regularly generated metrics should be introduced to track 
certain types of performance, such as delivery and coverage. While certain facets of decision support will 
require more careful analysis, the more routine aspects that relate to satisfaction of customer requirements 
should be amenable to simple self-monitoring that could be publicly available as product metadata. 
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6.0 Acronyms 


AVHRR Advanced Very High Resolution Radiometer 


AVISO 


Archivage, Validation et Interpretation des donnees des Satellites Oceanographiques 
(French data archive) 


CADRE Crop Assessment Data Retrieval and Evaluation 


CNES 


Centre National d’Etudes Spatials (the French space agency) 


DORIS 


Doppler Orbitography and Radiopositioning Integrated by Satellite 


DSS 


Decision Support System 
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ECMWF 

European Centre for Medium-Range Weather Forecasts 

FAS 

Foreign Agricultural Service 

GDR 

Geophysical Data Record 

GIMMS 

Global Inventory Monitoring and Modeling Studies 

GLAM 

Global Agricultural Monitoring 

GRLM 

Global Reservoir and Lake Monitor 

GSD 

ground sample distance 

GSFC 

Goddard Space Flight Center 

IGDR 

Interim Geophysical Data Record 

MODAPS 

MODIS Data Production System 

MODIS 

Moderate Resolution Imaging Spectroradiometer 

NDVI 

Normalized Difference Vegetation Index 

PECAD 

Production Estimates and Crop Assessment Division 

RGB 

Red Green Blue 

TMR 

TOPEX Microwave Radiometer 

USD A 

U.S. Department of Agriculture 

USDA-FAS 

USDA Foreign Agricultural Service 

USGS 

U.S. Geological Survey 

V&V 

Verification and Validation 
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